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B8TRACT 

' Tha papar praaanti an avaluation daiign for naaiuring 
tha af factivanaii of ipacial aducation prograna for handicappad 
^raichoolara. Raaaoni for not ialacting traditional avaluatVon 
daaigni ara notad, and tha uia of a "changa-iadax" ia aupportad 
inataad. A valua-addad analyaia vaa aalaetad to datannina tha valua 
addad by tha prooran baypnd that vhich would hava baiin axpactad (dua 
to Maturation) without tha program. Factor a iuch aa aax, raca/athnic 
group, handicappinig condition, and family incoma wara alio takan into 
account. Analysaa of child charactariatica and program 
charactaristica (cantar or homa-baaad, occupational, apaach, and 
phyaical tharapy, and attandanca) ara urtdarway and will ba complatad 
uilng multipla radrasaion tachniguaa. An appandad chart liata atapa 
in tha valua-addad analyaia. (CL) 
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APPROACH V THE PRESCHOOL EVALUATION PROJECT 
TO MEASURING .V BPPECTIVENB88 OP EARLY INTERVENTION^ 

Uthlttn H. Htbbtitr 
MontgoBtry County Public School! 
Rockvlllt, Maryland 

Tha purpoaa of thia part of our praaantation la to daacriba tha approach «• 
hava aalactad to anawaring quaationa about tha aff activanaia of apacial 
aducatlon prograai for handioappad praachoolari. Prograa avaluation in nany 
paopl a*a ainda tranalataa to axaalning tha iapact ol tha program on tha 
participanta. Tha narrovnaaa of thia viav hai baan Juatifiably critlcipad 
by thoia who faal tha iaaning of tha phrapa ahould1>a axpandad to includa^ 
othar kinda of infoniation collecting effort a auch aa.>aada aaaaaamanta or 
an axanination of a prograa*a prograaa toward it a objactiva (Shaahan i 
Gallaghar, 1983). Thaaa othar kinds of inloraation can ba vary halpful to 
progran adainiatratora and aarvica providara who want to iaprov^ thalt 
prograa. Unfortunately, whan funding daciaiona ara baing aada about 
prograaa for praachool handicapped children, prograa evaluation doea reduce 
to that one narrow queationt 

Do prograaa and aervicea hava a positive effect on thoae children who 
receive thaa? 

Thia project ia funded by the Mary laud State Pepartaaqt of Education and. 
while the atate ia ^ntereated in learning iny nuAber of intereating thinga 
about prograaa for praachool handicaj>ped Children, the inf oraation aoat 
needed ia data for the atate legialatura. Maryland ia one of tha few atatea 
that aandatea apacial education aervicea froa birth, and oae geti the 
iapreaaion that the continued exia ten ee of the aandata ia alwaya in 
jeopardy. The purpoae of the project then waa to provide the atate with 
data on the efficacy of early intervention with handicapped children froa 
birth through aix. 

Deteraining Effectiveneaa. 

(here are a nuaber of poeaibla approachea to deteraining prograa 
^ff.Mveneaa. One approach, a textbook approach in that aenae that it 
aeatff ill the aethodological criteria for a good evaluation, ia to begin 
with a aitable group of children, randoaly aaaign aoae to one or acre 
treataenta/prograaa and give no treataant to the reaainder. Obvioualy, the 
exiatence of a control group in a atate which aandatea aervicea froa birth 
la legally, not to aention ethically, out of the queation. 

A aecdnd approach providea everyone with aervicea but randoaly aaeigna 
children to different groupa ao relative aaounta of growth can be coapared 
accoaa the groupa. Thia approach ia alao legally inconaiatent with placing 
children in prograaa aoat appropriate to their needa. 
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What wt had £pr tha Mklngi of our •valuation waa tha praachool handlcappad 
population of Montfohary County. Tha county aarvaa approximataly 600 auch 
ehlldran in a variaty of progran and aarviea arrangananta including cantar 
and homa-'baaad aarvieaa; catagorical privata and public prograna for a 
variaty of handicapping conditiona auch aa l«nguaga dalaya or Vipual 
lapairaanta; and itinarant apaiich aarvlcaa for ehlldran with slid apaach or 
languaga dalaya. Tha ehlldran vara placad in thair programa according to 
tha county'a placananta procadurap, and it waa our taak to datarmiha whathar 
or not thoaa prograna had a poaitiva lapact on tha childran'a davalopnant. 
Givan that wa could not do tha aoat aathodologically daairabla kind of 
ayaluation, wa wara laft ifith aavaral othar choicaa. Ona po«ai1>la approach 
would hava baan to compara tha progtaaa of tha handlcappad ehlldran to that 
of tha ttorn group on aotta kind of a atandardiiad taat. Thia typa of an 
avaluation aakaa atataaanta Ilka "Tha ehlldran aada aix aontha gain in aight 
aontha tiaa." Tha firat problea with thia approach la daciding what aaount 
of gain eonatitutaa an af factiva prograa, or to atata tha problaa anothar 
way, how auch gain do you raally axpact tha ehlldran to aaka? Six aontha 
tain in aight aontha for a aavaraly handlcappad child aay ba avparb; aix 
aontha gain for a aildly languaga iapairad child aay ba tha kind of prograaa 
tha child waa aaking without tha prograa^ Tha aacond problaa with ualng a 
nora group aa tha atandard of coapariaon la that tha taat*a noraativa data 
waa baa ad on a group of non-handicappad ehlldran; it»a applicability to 
aaaauring tha growth of handlcappad ehlldran la quaationabla. 

A laat approach, tha approach wa hava aalactad, involvoa coaputing aoaa kind 
of a "changa indax". Holary and Bailay (1984) Hat a nuabar of auch changt 
indlcaa. Tha baaic concapt la to uaa what you knpw about tha child frba tha 
prataat or firat taatlng to projact whara tha child ahould ba functioning 
any nuabar of aontha down tha road. For axaapla, a child who la 24 aontha 
old and functioning at tha 12 aonth old. laval prior to intarvantion would ba 
projactad to ba functioning at tha 18 aontha old laval at 36 aontha if all 
conditiona atayad tha aaaa. If tha child la functioning at a highar laval, 
than tha diffaranca la attributad to tha affactiva program. Varibua growth 
rates ara Indicatad in FigMra 1. 

Aa Wolary (1983) pointa out, thia kind of analyala aaauaaa that tha child»a 
prctaat aga la a rafl action of tha child*a rata of davalopaant. To tha 
axtant that rataa vary and tha child la prataatad at a particularly "alow** 
tiaa, tha projactad rata of growth la inaccurataly low. Tha child aay ahow 
a aubatantial aaount of gain batwaan pra and poattaat dua antiraly to a 
dayalopaantal growth apurt* An aoalyaia which uaaa a changa indax, howaVar, 
will incorractly attributa that growth to participating in a prograa. Thia 
typa of aituation la illuatratad by tha laat graph in Pigura 1. 

Approach of tha Praachool Evaluation Projact 

Tha particular typa of changa indax wa hava alactad for thia projact la 
callad -valua-addad- analyala (Bryk 4 Halabarg, 1976; Bryk, Stranio « 
Waiabarg, 1980). Tha goal of tha fnalyala la to dataraina tha "valua^addad" 
by tha prograa abova and bayond that which would hava baan withbut tha 
prograa. Tha growth ahown by a child batwaan prataat and poattaat la 
partitlonad into two piacaat 

0 tha aaount of growth dua to aaturation 
0 tha aaount of growth dua to tha prograa. 



/ 



/ 



litiTw ""i )^ i« dont by pro jte ting vhtrt tht child 

vIioV'.J'Ah" ITS'? * icort baitd on tht rttultt of tht prtttttlhg. 

■•thod li «»ot pront to trror dut to dtvtlopmtntal iputti 
M""!** ""P*^'** tht tntlrt «rovp or iubgrou?i o£ 

ehlldrtn through • rtgrtiilon tquttlon rathtr thtn for tteh child 
Individually through tht uit of a ratlQ. I havt outllntd tht t;tchrtlcal 
Ik/A 2 your handouti. I will vary brltfly go through 

thoit attpa. It la a coBpllcat;ad atatlatlcal procadura but It la alfo 
undaratandabla avan without a background In atatlatlca. " ai-o 

IiU!!'I*'!l^f! •"•^y"*« ^"^^n* "ith tha projact la allghtly aora 

Tlltlr ! PtM-ntad aboya bacauaa In Stap 1 wa IncorporataS a Lbar 

i U >»yP»'»»«"««0 «i«ht ba ralatad to davalopmantal growth In 
our population. Thaaa factora warat k -.".afowtn in, 

0 ■ aax 

0 raca/athnlc group (whlta, black, Hlapanic and othar) 

o handicapping cbndltlon (apaach and languaga Inpalred. nultl- 

handlcappad, othar) 
0 family incona (low Incona, not low Incona) 

Baaad on thalr prataata acora, tha analyala calculatad a dlffarant growth 
rate for chlldran who dlffarad on thaaa charactarlatlca. For axampla. a 
black,, apaach and languaga Ittpalrad mala fron a non-low Incoma family had a 

dlffarant growth rata from a whlta, multfl-handlcaopad famala from a low 
mcoM zamlly« 

Tha valua-addad approach allowa ua to dttarmlna tha axtant of gain 4ua to 
program participation In aach domain of tha BDI for tha 123 chlldran with 
complata data^ Tha amount of gain In aach domain waa than taatad to aaa la 
:i4^** •Ignlflcantly dlffarant from 0. I'm not abla to ahara tha raaulta of 
thla analyala with you bacauaa tha raport haa not yat gona to our Board of 
BducatlQii. Purtharmota, I want you to know that wa ara Intarbratlng our 
flrat y^ar raaulta vary cautloualy bacauaa tha avaraga amount of tlma 
^^..V/^^'^'JV poattaat waa only 4.5 montha. Tha aacond round of 
poattaatlng with thaaa chlldran will taka placa at tha and of thla achool 
yaar and thaaa raaulta ahould ba conaldarably mora maanlngful. 

What Factora ara Aaaoclatad with Program Galna? 

our analyaaa did not atop with aaklng wara thara galna dua to program 
parltlcpatlon. Wa alao aought to datarmlna which chlldran and which program 
factora wara aaaoclatad with tha largaat amount of gain dua to program 
participation. Tha Initial modal wa ara exploring la rapraaantad In Figurt 
3 of your handout a. Tha child charactarlatlca balng examined art t 



0 aax 

0 race/ethnic group 

0 family Income 

0 age 

0 handicapping condition 



ERIC 



Th«>rogr«m charactarlttlet arai 

0 cantar or homa-baaad aoda of dallvary 
, 0 Quabarof ainutai in program 
0 , aaount of iipaach itharapy 
0 amount of phyalcal tharapy 
0 amount of occupational tharapy 
0 attandanca 

Thaaa anklyaaa will ba abla to anavar quaatlona Ilka "did boya maka mora 
prograaa in programa than glrla?" or "waa tha amount of gain in axpraaalva 
Itnguaga dua do program participation ralatad to tha amount pf apaach 
tharapy racaivad?" Thia anilyaia ihvolvaa tha uaa of multipla ragraaaloii 
with tha -valua-addad" aa tha dapandant variabla and tha factora liatmd 
abova aa indapandant variablaa. Unfortunataly, I cannot ahara tha raaulta 
irlth you but avan if I could I would ha va to warn you that thay ara baaad on 
a vary abort amount of tima in program. Tha data wa ara about to collact at 
tha and of thia yaar will ba conaidarably mora halpful in anawaring thaaa 
quaationa. , • 

pna axtramaly important faatura of tha modal which ia miaaing la datailad 
Information on what ia. actually happaning in claaarooma with tha childran. 
For inatanca, Indicatora auch aa tha amount of ^ima tha child ia working 
with an adult, with othar childran, or alona would ba moat halpful in 
axploring tha talationahip batwaan galna and atrvicaa. Datailad information 
About aarvica provlaion ia omitted from thii modal bacauaa tha only way to 
gat thia kind of information ia through claaarooa obaarvationa which ara 
vary coatly data to collact. Aa important aa auch information may ba to 
explaining prograaa, thara waa almply no way wa cbuld afford to collact it. 

Concluaion 

In cloaing tha topic of our approach to af f activanaaa, I would lika to 
ramind avaryona that thia typa of an avaluatipn daaign ii far from parfact* 
Aa you may hiiv« noticad, tha typa of approach wa hava baan f oread to uaa ia 
light y«ara away from a control group, randotf aaaignmant daaign. Tha 
litaratura in thia araa haa of tan criticiiad avaluationa Of programa for 
handicappad praachoolara hava many aarioua aathodological flawa (Tindal. 
1985; Whita, Maatropiari 4 Caatro, 1984). Tha many problama of paat 
avaluationa damonatrata that it ia vary difficult to do a good evaluation 
thia araa^ Poaaibly, it ia tima to racogniia that tha old notiona about 
program evaluation do not hold for thia populationa — or at laaat, thay 
don t hold at thia particular point in tima and right now happana to ba whan 
thoaa holding tha puraa atringa ara aaking tha quaationa. Ona aolution ia 
not to do avaluationa at all if all tha crltaria for a tight evaluation 
cannot be met; auch a aolution will hardly ba aatiafying deda ion-maker a. 
Another aolution, aa auggaated by Strain (1984) ia to ahif t the focua of the 
queation from "ia intervantion af factiveT" to "whet ia the optimum laval of 
changa poaaibla?" Our aolution ia to try at aoaa level to anawer both of 
thoaa quaationa. We want to know whether the childran hava ahown more 
growth than they would have without the program, but we alao want to know 
the coQditiona under which childran ahow the moat growth. 
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FlCUllE 2 
Sttpi in V«luc-Add«d Analyilt 
PMiTMs - emnh i»9 to MturatlOD ♦ grevth iXu^ to prograi participation 

iS2£ gxaaplt 
1. Itsrtii prtttit on at«. Prttait - .8 Age 



i- 



2. 
3. 

4. 
5. 



Use cotffldtnt fron the rtgrtMlon 
ai tha growth rata prior to pratait. 

Multiply rata by tha anount of tiaa 
in prograa to datanine tha aaount of 
growth dua to, aaturatiOR. 

Subtract prataat froa poittait to 
^ataraina tha total ammnt of growth. 

Subtract growth dua to Mturation fron 
total growth to dataraina growth dua to 
prograa participation. / 



Coafficiant • .8 

.8 X 10 aonthi ■ 8 points 

36 - 24 - 12 
12-8-4 point! 



Onmth 4tta to pngraa partlcipatioa - 4 pelau 



Coaputing Coafficianta for Diffarant Groups 



1. Ragraai pratait on aga, laz, raca, 

handicapping condition, faaily incoaa 



.2 f_. Uil, •^orrtfitf 4 -£0««iciant:i- which 
dapandi on child *• charactariitici. 



Continua as abova. 



Pratast - .8Aga -.3Multi- 
handicappad 4 
.INala... 

Coafficiant|QiB|^i« 

.8 - .r+ .1 ■ .6 

C6afficiantMHf|aala " . 

.8 - .3^-.r • .5 
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FIGURE 3 

What Factors Arc Aicodctcd with Gains 

Due to Prbgraa Participation 
for Handicapped Prcachoolcrg? 





CHILD FACTORS 
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ract/tthnlc group 
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faally incoac 
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handicapping condition 




PROGRAM FACTORS 

0 canter or hoao-baatd 

0 nuwbtr of ■Inutts in prograa 

0 amount of apttch thtrapy 

0 aaount of phyaical tharapy 

0 asount of occupational thtrapy 

0 attandanct 
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